Repository navigation
ci: add a Commit-Boost check to the kurtosis battery - #708
MegaRedHand wants to merge 23 commits into
Conversation
The builder PR stack (#694 to #705) lets ethlambda's beacon node and validator client build blocks through a MEV-Boost compatible builder, but nothing exercises it against a real relay and builder. Commit-Boost's PBS module is a drop-in MEV-Boost replacement, so a devnet with `mev_type: commit-boost` covers the whole path. The check reads the ethlambda services' builder metrics instead of running assertoor, because the base battery's assertoor tests currently fail on an unrelated spec-endpoint gap. It passes once registrations are forwarded, the ethlambda beacon node serves a builder payload, and the ethlambda validator client proposes blinded blocks both through that node and through lighthouse. The new `commit-boost` job runs on manual dispatch only: main's image has no `--builder` or `--builder-proposals` until the stack lands. It has its own ethereum-package pin, which needs the fork's builder-flags branch pushed. The relay, builder and PBS images are pinned by digest since the package defaults are floating tags.
A contributor running the check locally needs to know how to get an image with the builder flags and that kurtosis will use it without pulling, or it looks like the run needs a published image. The fork pin also now has a PR to point at instead of an unpushed branch.
🤖 Codex Code ReviewLooks good overall. I did not find consensus-layer logic changes here; this PR is CI/tooling-only, so fork choice, attestation processing, state transition, XMSS, and SSZ risks are not directly touched.
If you want, I can turn the first three points into concrete patch suggestions. Automated review by OpenAI Codex · gpt-5.4 · custom prompt |
🤖 Claude Code ReviewReview of PR 708: Commit-Boost kurtosis checkThis is a CI and tooling change with no consensus code. It looks sound overall. I read the diff but did not run the workflow or the script, so the points below are from reading only. Good
Points to consider
No blocking issues. Automated review by Claude (Anthropic) · sonnet · custom prompt |
🤖 Kimi Code ReviewI'll review this PR, which adds a Commit-Boost builder test battery to the Kurtosis/Assertoor CI workflow. The changes are primarily in CI configuration, documentation, and a shell test script — no consensus-layer Rust code is modified. General AssessmentThis is a well-structured CI addition with good documentation. The shell script has several issues that could cause flaky or incorrect behavior in CI. Detailed Findings1. Shell Script:
|
| Priority | Item | Location |
|---|---|---|
| High | Fix notify job to depend on commit-boost |
.github/workflows/kurtosis_assertoor.yml |
| Medium | Fix awk integer parsing in value_of |
scripts/check-builder-blocks.sh:63 |
| Medium | Cache metrics to avoid race conditions | scripts/check-builder-blocks.sh:88 |
| Low | Add kurtosis availability check |
scripts/check-builder-blocks.sh (top) |
| Low | Verify/fix date comment | commit_boost_params.yaml:44 |
| Low | Add TODO for fork re-pin | .github/workflows/kurtosis_assertoor.yml or README |
Automated review by Kimi (Moonshot AI) · kimi-k2.6 · custom prompt
A local run did not start. The newest reth-rbuilder image parses only enode bootnodes while the package passes ENRs, and Commit-Boost's PBS requires SECONDS_PER_SLOT, which the genesis generator no longer writes. The first is fixed in the fork (PR #23), so the package pin moves to that commit. The second is worked around by giving Commit-Boost an explicit chain through `commit_boost_config`; its `chain_id` must be a hex string. The README lists both workarounds so they can be dropped when upstream is fixed, and explains the harmless "Builder is not answering" logs at startup.
A local run registered every validator but never got a builder block: rbuilder submits no bid for a block without transactions, and the enclave had none, so the relay held no bids for any proposer and every getHeader answered 204. With transactions flowing, the same enclave served a builder payload within a few slots, and the ethlambda validator client signed the blinded block that the relay delivered. spamoor runs with the package's default spammers (transfers and blob transactions), pinned by digest like the other images. The README also explains the no_finality fallbacks a run counts before the chain first finalizes.
With spamoor's defaults (a chain-load group, 200 transfers and 6 blob transactions a slot) the relay's getHeader slowed to Commit-Boost's 950 ms timeout, so most proposals got no bid, and in 40 minutes the ethlambda validator client behind lighthouse never received one. 10 transfers and 1 blob transaction a slot keep bids coming without that slowdown: a local run met every condition about 12 minutes after the enclave started, and the relay delivered a builder block in 14 of the 32 slots of epoch 1. The package's own 100 Uniswap swaps a slot, which it adds whenever a builder runs, stay on; they are not configurable.
#713 locked every workflow's actions in .github/workflows/actions.lock, after this job was written. Its checkout@v6 and upload-artifact@v7 no longer match the v6.1.0 and v7.0.2 the lockfile pins for this workflow, so verify-lockfile fails and the enforced lockfile refuses to start the workflow at all (startup_failure on the PR's Kurtosis Assertoor run). Use the same refs as the assertoor job beside it, including the `$/` same-repository form #713 gave the free-disk action.
…s explicit From the automated review of this PR: - Counter values are summed and truncated in awk, then printed with %.0f, so a large or exponent-notation value (1e+06) reads whole and a value below 1 never meets a >= 1 condition. - Each service's metrics port must resolve before polling starts. A renamed service, a reordered participant list or a missing kurtosis/curl now fails at once with exit status 2 and kurtosis's own message, rather than reading as "not met yet" for 40 minutes. - One evaluation scrapes every service once, so the conditions on one service read the same snapshot. - A service that does not answer is reported as such, not as a 0. - failed.txt is rewritten after every evaluation, so a cancelled run still says what was unmet. - The address kurtosis prints is reduced to host:port, whatever scheme or path surrounds it. Checked against a stub kurtosis and local metrics servers, under macOS's bash 3.2 and under ubuntu:24.04's bash 5.2 with mawk; shellcheck is clean.
…the Commit-Boost job notify reports pushes only and the commit-boost job runs on dispatch only, so adding the job to notify's needs today would wire up nothing. The note sits where the event condition is dropped, which is when it starts to matter.
|
/review |
🤖 Codex Code Review
Shell syntax validation passed; the devnet was not run. Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Gemini Code ReviewThis PR looks good. The workflow conditions, Kurtosis parameters, metric-polling script, and documentation are well-structured and handle failure and cleanup paths cleanly. A few highlights:
Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 Claude Code ReviewReview of PR 708: Commit-Boost check for ethlambdaThis PR only touches CI and tooling; no Rust or consensus code changes. The tooling is well built: it pins images by digest, passes Likely bugs
Robustness of
|
🤖 Grok Code ReviewI'll compare the new Commit-Boost job and check script with the existing Assertoor workflow so the review is grounded in how this repo actually runs Kurtosis. Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 DeepSeek Code ReviewReview summary: this is CI/testing tooling only, no consensus-layer or Rust code. Overall the approach is reasonable, but there are a few concrete issues.
No consensus-layer correctness, SSZ, XMSS, or Rust memory-safety concerns in this PR. Automated review by DeepSeek · deepseek-v4-pro · custom prompt |
🤖 Kimi Code ReviewNo blocking correctness or security issues found. The workflow gating handles all three battery choices correctly, and the pinned package commit/container digests, Minor robustness suggestion:
The dispatch-only limitation means this does not yet provide push/PR regression coverage, but that is clearly documented, including the required follow-up to enable the job and update Slack notification dependencies. Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
…-slashing-protection Since #734 the ethlambda validator client refuses keys with no slashing-protection record, so both ethlambda VCs exited at startup and kurtosis rolled them back. network_params.yaml has passed the flag since then; this file now does the same, for the same reason: genesis keys in a fresh directory have never signed.
The step piped kurtosis run into tee under bash -e without pipefail, so it took tee's status: a Starlark error passed the step and surfaced only as missing services in the next one.
🤖 Claude Code ReviewPR 708 review: Commit-Boost check (CI and tooling only)This PR touches no Rust code, so the consensus-layer items (fork choice, state transition, XMSS, SSZ) don't apply. The change is well documented, and the shell script is careful: it uses pipefail, takes one snapshot per evaluation, fails fast with exit 2 on setup errors, and rewrites Likely blocker
Workflow
|
🤖 Codex Code Review
Shell syntax validation passed. No Rust or consensus implementation changes are included. Automated review by OpenAI Codex · gpt-6.1-sol · custom prompt |
🤖 Gemini Code ReviewLooks good to me! The workflow wiring, Kurtosis parameters (with digest-pinned upstream images), metric polling script, and README documentation are all clean and well-structured. Automated review by Gemini · gemini-4-argon · custom prompt |
🤖 DeepSeek Code ReviewThe PR is entirely CI/tooling (workflow, YAML params, bash check script, documentation). No Rust protocol code is touched, so consensus-critical correctness is not affected. The bash script is generally well-written and robust. Below are the specific issues found.
|
🤖 Grok Code ReviewThe Commit-Boost wiring is careful (digest pins, fail-fast on bad service names,
metrics_address() {
local addr
addr=$(kurtosis port print "$ENCLAVE" "$1" metrics) || return
addr="${addr#*://}"
printf '%s\n' "${addr%%/*}"
}
...
curl -sf --max-time 10 "$addr/metrics" 2>/dev/null || trueCurl's parser treats the colon as a scheme ( Also worth fixing while you are there:
Automated review by Grok (xAI) · grok-4.7 · custom prompt |
🤖 Kimi Code ReviewNo blocking correctness or security issues found. The workflow dispatch conditions correctly handle One non-blocking diagnostics improvement:
Otherwise, the changes look well documented and appropriately isolated from the existing Assertoor battery. Automated review by Kimi (Moonshot AI) · kimi-k3.1-pro · custom prompt |
Adds a Commit-Boost check to the kurtosis battery: ethlambda builds blocks through Commit-Boost's PBS module, the MEV-Boost replacement that speaks the same builder-API.
What it runs
tooling/kurtosis-assertoor/commit_boost_params.yaml:mev_type: commit-boost. ethereum-package then adds a flashbots relay, an rbuilder with its own lighthouse, and one Commit-Boost PBS sidecar per participant.GLOAS_FORK_EPOCHspec mismatch (README, "Known failure").scripts/check-builder-blocks.shpolls Prometheus metrics until all of these hold, or 40 minutes pass:lean_beacon_builder_registrations_total{outcome="forwarded"} >= 1lean_beacon_builder_payload_source_total{source="builder"} >= 1produceBlockV3served a builder blockethlambda_validator_builder_registrations_sent_total >= 1,ethlambda_validator_blinded_blocks_proposed_total >= 1ethlambda_validator_blinded_blocks_proposed_total >= 1The script writes each service's builder metrics and the unmet conditions to the results directory on every exit.
Dispatch only, for now
main's) has no--builderor--builder-proposalsuntil the builder stack (feat(beacon): builder-specs containers and the fulu blinded block's state root #694 to feat(validator): propose builder blocks (fulu) #705) lands. The newcommit-boostjob therefore runs only onworkflow_dispatchwithbattery: commit-boostorboth, against an image built from that stack.main, drop the event condition so the check runs on push too.Depends on
a15e7df): adds ethlambda toapply_external_builder_flags.reth-rbuilderimage refuses the ENRs the package has passed since its 08-05 refactor.COMMIT_BOOST_PACKAGE_REFis feat: receive and decode gossiped attestations #23's commit, separate from the base battery'sETHEREUM_PACKAGE_REF. Re-pin to the merge commit once feat: receive and decode gossiped attestations #23 lands.Upstream workarounds
SECONDS_PER_SLOT, which the genesis generator no longer writesmev_params.commit_boost_config(chain_idmust be a hex string)Tested
shellcheck,actionlintandyqpass.ae1a3d2f) met all five conditions about 12 minutes after the enclave started.